Source Inclusion in Synthesis Writing: An NLP Approach to Understanding Argumentation, Sourcing, and Essay Quality

Autor/inn/en	Crossley, Scott; Wan, Qian; Allen, Laura; McNamara, Danielle
Titel	Source Inclusion in Synthesis Writing: An NLP Approach to Understanding Argumentation, Sourcing, and Essay Quality
Quelle	In: Reading and Writing: An Interdisciplinary Journal, 36 (2023) 4, S.1053-1083 (31 Seiten)Infoseite zur Zeitschrift PDF als Volltext Verfügbarkeit
Zusatzinformation	ORCID (Crossley, Scott) Weitere Informationen
Sprache	englisch
Dokumenttyp	gedruckt; online; Zeitschriftenaufsatz
ISSN	0922-4777
DOI	10.1007/s11145-021-10221-x
Schlagwörter	Writing Skills; Cognitive Processes; Essays; Cues; Writing Evaluation; Natural Language Processing; Persuasive Discourse; Semantics; Information Retrieval; Citations (References); Plagiarism; Predictor Variables; Computer Software; Computational Linguistics; Evaluators; Scores + Suchen Sie Ihr Suchwort? Writing skill; Schreibfertigkeit; Cognitive process; Kognitiver Prozess; Essay; Aufsatzunterricht; Stichwort; Natürliche Sprache; Persuasion; Persuasive Kommunikation; Semantik; Citations; Zitat; Plagiat; Prädiktor; Linguistics; Computerlinguistik
Abstract	Synthesis writing is widely taught across domains and serves as an important means of assessing writing ability, text comprehension, and content learning. Synthesis writing differs from other types of writing in terms of both cognitive and task demands because it requires writers to integrate information across source materials. However, little is known about how integration of source material may influence overall writing quality for synthesis tasks. This study examined approximately 900 source-based essays written in response to four different synthesis prompts which instructed writers to use information from the sources to illustrate and support their arguments and clearly indicate from which sources they were drawing (i.e., citation use). The essays were then scored by expert raters for holistic quality, argumentation, and source use/inferencing. Hand-crafted natural language processing (NLP) features and pre-existing NLP tools were used to examine semantic and keyword overlap between the essays and the source texts, plagiarism from the source texts, and instances of source citation and quoting. These variables along with text length and prompt were then used to predict essays scores. Results reported strong models for predicting human ratings that explained between 47 and 52% of the variance in scores. The results indicate that text length was the strongest predictor of score but also that more successful writers include stronger, semantically-related information from the source, provide more citations and do so later in the text, and copy less from the text. This work introduces the use of NLP techniques to assess source integration, provides details on the types of source integration used by writers, and highlights the effects of source integration on writing quality. [For the corresponding grantee submission, see ED619918.] (As Provided).
Anmerkungen	Springer. Available from: Springer Nature. One New York Plaza, Suite 4600, New York, NY 10004. Tel: 800-777-4643; Tel: 212-460-1500; Fax: 212-460-1700; e-mail: customerservice@springernature.com; Web site: https://link.springer.com/
Erfasst von	ERIC (Education Resources Information Center), Washington, DC
Update	2024/1/01